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COMPARISON OF THE EFFECTIVENESS OF SIX MODELS IN . 
• " FORECASTING THE STUDENT DEMAND ON ACADEMIC DEPARTMENTS, 

SUMMARY ' . " • ^ 

' J An accurate .forecast of the student demand by level .on the academic 
departments of an institution Is vital for budget and financial planning ^ , " 
decision-s, for faculty workload scheduling and for physical facility plan- 
ning. Many me.thods have been, used to'forecast this demand ranging, from 
''^ste^of your^nts" guessing to highly complex 'computer models.^ . 
" V TmTT^)^ project studied six basic models for forecasting student 
demand at vari^jlevels of sopjiisticatiof and complexity. . The models studied 
included judgeffi^t only, a ratio model, a Markov model and a -combination 
model^ In addition, the dimension of expert judgement- was- combined, to one 
model to determine the valwe of the additional input. 

The model wi1:h the. expert judgement added was the best model based on 
' the criterion of least-error using several error analysis indices. The 
simple models gave as good as or better forecasts th^an the more complex 
models' Jsing^ the same least-erjor criterion. Also,, using a cost-eff ective- 
[jless criterion, theAl^Ple models were again superior to the costly, sophis-, 
ticated model s. ., . - . 



SECTION I-I: DESCRIPTION OF STUDY 



PROBLEM STATEMENT 

Ten-year student enrollment forecasts have two 'primary uses— financial 
planning and facultj' scheduling. Ong^of *the basic inputs to a university 
budget or long ranc^e'plan is an accurate prediction of the student load by 
level on each academic department. Student demand by level is so vital be- 
"cau.se cost per stud'ent varies widely by department and also by level with- 
in ' a department. Recent studies on the .cost of instruction at the University 
oV Utah29 have- shown that cost per, student differs by as much^ as a factor of 
14v5. between departments and by as much as a 'factor of 46.6 between levels in- 
the s^me department. • . • . " «©' - 

In addition to budget and financial planning decisions, another j'mpor- , 
•tant use of student demand data is for planning 'faculty workloads and for \ 
scheduling faculty, classroom and advising assignments. •• \ 

Many complex factors affect the student demand on departments such as ■ 
changing student expectations, judgments concerning career opportunities, 
and varying" economic and political conditiong/^These fluctuating student^ 
attitudes make 'the forecasting of enrjollment by department increasingly 
difficult. To compound the problem, these dynamic changes are occurring at 
a- time when budgeting constraints in both the short and long run are demand- 
ing more accuracy and when scheduling each faculty member to -obtain -the-eemplete 
utilization of his talents i,5 critical. ' 



BACKGRQUJ1D > 

Many attempts- have been made over the ye.ars to forecast student' en- 
. . 'J ■ • 

rollments at universities. Some early methods were for university budget 

makers to make an educated guess, on next year's enrollment. For small 
schools with only a few departments and limited students', ;these rough esti- 
mdtes'>were accurate enough. 

As schools increased ir\ the number of student^ and in the n,umber of 
l€^i^ehts, the budget makers had an increasingly difficjlt time making 
j(ccurate forecasts. In addition, the budg^'ting and plannihg hori'zon that 
the administrators were concerned about kept expanding into the future. So 
the student forecasting problem was compounded beyond the limits qf t-he 
administration to handle by the' "seat of their pants" or by their judgement 
alone. 

' The next development was to use some mathematical technique or model 
for forecasting student enrollments. The models that have proven most suc- 
cessful fall into the following' /i-ve byroad categories: 

1 . -Ratio Techniques ' : ^ 

2. Regres^ipn Analysis 

3. Markov Models - ' . 

4. Simulative, Branching Network and Prqgramming 
Techniques 

SNvQombi nation Models ' ^ 

A brief discussion^ each of these forecasting methods as applied 
to student enrollments /follows in the next sections. , . ^ 

"CATEGORIES OF STUDENT PROJECTION MODELS 

Ratio Techniques ' , - l/ 

Ratio' techniques' are generally based'on the assumption that 9 ratio 



adequately describes the probability of passihg from one state or classi- 
fi/Cation to another. These "states'" usually refer, to a ^ategory of a' . 
classification variable such as student level (undergradLte) , major (history) 
or status (continuing). Some of^the models uSing the ratio method as • , ^ 

basis for forecasting "follows: Cohort Survival Teefmi^quer-^' . • Class Rate 
^Progression Technique, and Simple Ratio Method* 



''^ Regression /^na lysis ^ ; ' . . * --^ 

Regression techniques are generally based on .the aissumption that the 
trends and relationji^ips^bserved in the past will continue in the future 

and tjiert'^the^'s^les are taken independently. Several 'model s ir\corporatihg 

I " '16 

tl^^s technique as part of a larger, system are'-as' follows: the CAMPUS Model 

developed by the Systems Research Group, th'e Michigan S'tate University Wddel," 

the Tulane University Model ,^ and the Peat, Marwick, Mitchell and Company 

CAP:SC Model. 27 The Trend Line Model ^^analyzed in the ACT research is of 

the regression Cype. The University of Utah model^ is basically a I1)near 

regrqsfion moddl . Thi/s model will be discussed, in more detail in a^subse- 

quent section. ' - ' . 

' ' Markov Models [ ' 

Markov models are generally based on the basic assumpti^on that 
transitions from one 'state\to another during an increment of time depend 
on the present state and arte\ independent of the past. The Markov model is a 
discrete time system charactier^zed by a collection pf probabilities 
(tra'hsition probability matr-ii\«). Each probability represent the likelihood 
'that a student will move from his, present stite to another state during the 



next time interval,. Another assumptioi|i of this model is that this transition 

. ■ ' • - ' " . .V ( • 

probability matrix remains cohstant over time. ; . 

' .' The Markov model is 'perhaps the most popular model of student flow' 
•^t the>resent time, 'at least in the literature.; Some .of the pi)b.l isl^ied rie^ 
ports of Markov models include the following:. Gani ,^ Young-and Aljjitfnd,32 
Oliver 01iver<and Marshall Marshall, Olive/r and Suslow/^Orwig, ~3ones 
and Lenni^ng,^^ State of Washington Johnson^ ^ and^^FLi>lerton .^^ 

As can, be surmised from the number of cj4r|nt models, the Markov 
model is very popular for foreclisting student enrol^nts. The basic 
Mar]/pvian, assumptions, however ; t^nd' to be counter-intuitive, as pointed out 
by Lovell.^'^ The stationarity assumption seems inappropriate in the dynamic 
educational sys/tem! Also the assu;iiption tljat future transitions are com- 
pletely indepetident of the past is questionable in the flow of students 
through the higher^, education system,. 



^ / Simulation, Branching. Metworic and Programming .Techniques 

other models can cociQeivab'ly be applied to, the student flow process, 
sud/as Monte Carlo simulations., lOcertain classes of branching,^ perca'lation/' 
pr/cesses,4 network flow theory^ and linear, non-1 inear and/dynamic pro- ' 
• ^^ramming techniques. 21' 

^ Most of these models have not as yet proven successful in representin^^r^,^^ 
/ student flow but they do offer some proinise for^uture research. 




Co mbination Models , < ^ 
] Several mo^ls us^d at institutes of higher. education for forecasting^' 
student-demand are/\con;bf nations of the above majcfr categories of techn?^s, 
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/ . • ^ y.:- -->:. J/ - ' ■ 

Thyu.C.L.A. !tiode1^° uses se^erartiprojeittion techniques starting witl^e^s- 
s/on methods to estimate the ntimber of students and then projecting enrol 
lent. demands using r?it-1o techniques^ 

The RensseUe^ M?)del2-uses a Markovian pric^ss for projections but denves 
the transition matrices using regression methods on previous year's data. ^ 
The modef is, in effect, a nonstationary Markovian model. 

* "The University of Colorado simulation of operations model (CUSIM)^ . 
includes a student flow model which is a mixture of cohort survival ratios 
and regression smoothing. , ' ' 
PROBLEM DEFirilTION ■ ' , ♦ . 

Thus, over the'years, new and innovative techniques, have beem employed^ 
to forecast student enrollments, each one addi-ng a new dimensioif of sophisti- ' 
cation and) complexity. Perhaps the b^st-.to-date model for predicting -student', 
load on departments is the pne that NCHEMS at WICHtMs currently testing, 
SFM-IA. Preliminary documentation''^ indicates that all the latest and proven ^ 
modeling techniques have been incorporated into th^^'iiQ.^newest model 9f student 



flow including such innovations as transition probabil ity'totj^ceg and Markov 
chains. ' » " • ' ' ' . . 

' In another recent pubTication,26 the American ^1 lege Testing Program.. 
(ACT) jej^ed a study comparing five methods for projecting enrolment, 
4.,..««««rrte1?(iing ratio, regression and two Markov models. The conclusion of this 
■udy was,, among other things,' that "simple and straight-forwar^ projection 
models would appear 4o be jtist as^^usdful as complex and sophisticated models." 

Thus the following questions are raised: I s^ the NCHEMS, model 5uf)erior for 
7orecas'1;ing |he student demand- on a department by level? 'Or is 'there anbther 
modeT that incorporates techniques in addition to the mathematical techniques 
that would improve the 'forecast? Is the additional expense of gathering the 
detail "data and running the complex model justified in terms of increased 
accuKacy.in projections? '-5- . ' .. . ' 



l»A 1./* \-T\ \^T^\'^ ^ { 



FQRECASTIT^G METHODS EXPLODED • . ' * , 

Thjer^ are many'diffeiTent ways in- which to describe 'the methods used* 
in devel6pTng forecasting models. 'A recent article'examines two aspects' 
of ftis probleiu--the method and the type, of infprmatjon.- The first deals 
With the method used to analyze the data ajad is labeled the -llsMbjective-" 
objectiV€'\dimen.sion. Thp 'second deals with the. type of information and 
IS labeled the "naive-caudal" dimension. 

' ' Subjective vs.. Objective Methods 

Subjective methods are tho^e in which the procei^s used' to obtain 
the forecasts has riot been well specified. These are the judgmental, in- 
tuitive, "seat of the pants" methods mentioned earlier. 

Objective methods are those in which the proce'ss used -£0 obtain the * 
forecasts has been extremely wpll specme^. These methods- lend themse.lves 
well to computer processing and are so tightl>y defined that other., researchers 
■can replicate the method and ^(^tain e»(actly^|he same' forecasts. 

Naive •'VS . Causal M'pthods " * 

Na'ive methods' are those wHich use data on only the dependent variable , 
(e.g., number' of students enrolled). - Typi-call/, an analysis is earned out 
to see whether the dependent variable shows any regularities over time. The 
time pattern is then- projected into the future as shown in Figure 1 (a). 

CausaT methods go beyond the "dependent variable to consider also 

. ' / 
* . \ ' 

variables' which may .cause changes^ft the dependent variable. An attempt is 
ma<le tp'det'e^^qffke what causal variables are important, then to forecast the ^ 
causal varia4^^^ to 'infer values lor the- dependent \^ariable 

on "the ba„^ o| Arte /changes in' the causa'l variables, this procfess is out- 
linedLMn Figure f(b) . '^^^^^^^^T^^^^^ assumptions -are that the causal variabl|| can 
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Figure 1 V--.AfrWustration of Naive , and 
^ Causal, Ft3recasting Methqd-s - 
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Key to Symbols: 

Y is the dependent variable 

X is the set of causal variables , 

h "is the number' of years of historical 

f is the number of years in the future 

-i „ ' ■ ■' ■ . ■ • ■■ 

t is the. present year 
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be m'easure^l^nd projected" rather,' accurate 




in- compaHson to , a projection-. ' 
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©f th^iependent variable- and, that thife rel atibf^hips wi 1 1 J^emain 

•- over timcv. , , " /' 

. ^ Theoretical Forecasting Models --j- .^ . ; 
"tonsider each 'pf the.s^!^ tHO aspects of th* forecasftnp prpblertKas- a , 
.dimension inIforeca.sting" model spa^'e, FigMfe*2. with subjective-objective ' 
on one axis and naive -, causal on the otheK ^Armstrong and' Grohman have 
labeled the intersection of the extreme points of these dimensions ar^. shown 

*ir) figure 2^ * ^ :\ r ' • 

Armstrong's and Grohman's paper -draws the following conclusions: 

1. Objective, methods are more accurate than subjective methods. 

' 2. • Causal methods are^ more accurate than na'ive methods, > ' ■ . 

,- 3. ,The ''superiority pf objective and caus^. methods increases" as 

the forecast- horizon increases. '\ ;• ' 

. 4. , Therefore, econometric -methods will^jproduce, more- accurate- Kong- ■ 

\ ;, - , rarvge forecasts thaTi may bri'obtainecfv from novice judgement, exr 
, V:*^ V • . , trapplation or expert judgefagnt. V, , . _ . 

l' \ in the.context of the above anal ygis, the early student forecasting 
j, methods-were "Expert. Judgement" wMle the vaA,ious mathematical models are » 

•>v, ' ' . ■ 

"^^Extrapolation." According to Armstrong and dl-ohman,, a superior lortg-term 
forecasting method would be a combination of th^^se methods,. They suggest an 
„ econometric model which wbul;d be extremely compl Icated dem'and lots of 
causal analysis, requjire>iy!olumes of cjata and still iiot guarantee that the - 
causal^ relationships c-ar^ be'/e'S^'^b'l ishecTI^Dr "studen* d^^^ on.departraents. ; 
A realistic- compromise frsm tH^'economietric method wouTd be a ma^hemkticaT 
extrapolation modified by -expei^l judgement. One such modal 'has already, been 
developed at^the. University of U|ah and is Used extensively in preparing 
' manatiement deci-sion tools for budget allocation and long rang? planning. 

i.-/' ... . ' ■ / 
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Figure 2'. -Some Theoretical Types Of 
Forecasting Methods • ' ' 
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UNIVERSITY OF' UTAH MODEL ' ■ ■ ■ , 

At the University of Utah, k^udent proje'ction model-^ has been developed. - 

that forecasts the student demand by level by departinent .for ten -years. The 
.aggregation levels considered by the; model . are lower 'Si vision, upper division 

and'gradtiate. The model uses a ten-year student credit hour (SCH) history as 

* 

a base. From this histori-cafl data both a ten-year linear regressiqn and a _ 
current four-year regression are made on each level for all departments'. These 
regressions are then extrapolated out ten years.' The average of;^the two end 
point years is fixed as one end of the' projection line and^the last actual year .is 
fixed as the other ehd.. A linear interpolationvis made between' these ^ points'. 
^Each department's SCH is forecast by this method, and the total BCH j"s- added 
together by level to determine the total university §CH by level. 

An independently derived studgnt headcount ten year total university forecast 
by level is al'so'' input to the model. Thi s' forecast takeS.into consideration such 
factors as predicted number 'of high school graduates, students infclination for 
university training, job market demand for college grad,uates, drop out rates, 
credit by examination and other, factors that affect student enrollment. This 
headcount forecast is converted to SCH on the basis of ah average/student course 
load by'leve]:. then' the departments are each Adjusted so that the, total aggra- 
gated „dep,artment -SCH by" level equals the converted university) headcount^SCH 

total by level. ' ' ' ■ , , 

The- program aUo has the pro vfs ion for adjusting the. individual departments 
,SCH projection 'by ..level tp allow, fo'f special knowledge^the university admini- 
sirators may have that is not refTected. in historical data or in gross, overall 
trends, such as the phasing out of- a department over theli^t three years, a 
physical space 1 i mi tafifeij^ that will bfe reached in three -years, a leveling off of 
a high cost pVogranv, etc. <rhis special knowledge-can be input ih one of three 
• ways,; actual StH- by level by yisiar,' selecting an end point level, far the 
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tenth yes'r.y:Or choosing the terj^ four year 'linear extrapolation as the pro- 
jection line.. Of the three methods, the first should, not be, oyerlookeJ in 
importance due to the significant flexibility it allows. ■ ^ 

-.A check on the validity of the data is'obtained by distributing the 
f^inalized projections to the 'department. chairmen -for comments Or amendments. 

The above technique of allowing knowledgable administrators to ■ ' 
alter cerlMiin of the mathematical projections is related to managemler^ by 
exception and ^ is called "^recasting by exception." This added dimensibn 
to mathematical forecasting has,»^m^ong others, the following advantages 
dver pure extrapolatioji: r ' . 

■ •„ 1". Sys\§m makes routine pi^ojectioh^ for each level of each , 
'department. At the University^of Utah, this, amounts to 
144; separate projections. " . ^ 

2; System makes fuller use of knowledge of trends, history 
.and other available data. - , 

"3. System is consistent ^d" pred-ictable in it^ judgements, 
removes pf-ejud^ce frdm forecasts. r ^ , 

4. ' Management can" concentrate efforts on. critil:al "problem areas. 

5. Allows crises and' critical problem' ereSis to-be analyzed and 
• ' '\ adjusted by concerned', knowledgeable people. 

..'•■', 

6. Enables spacial 'knowledge of planned changes .to -.be incorporated 

• into the forecast, such as increasing the freshman class of . 

• the medical school by 75 students next fall. , - . ^ 

7 Simulates communication between various segments of the 

university-administrators, staff, deans, department chairmen ■ , 
and faculty..*^ 



P.URPOSE ANB VALUE OF THE STUDY , ' <^ 

" The purpose of this study is to compare' the accuracy oi; the forecast 
of student demand by, level on academic departments' using six basic methods - '^^^ 
judgement, ratio model, moving averages, regression model, Markov model and a 
combi-nation model. (Student demand on a department is measured in terms nf 

student credit hours SCH In add.ition the researerher investigated the valae 

' ^ . ' ■■ • • 

-11- 



of coml>i.ning expert j^'udgement with the ma them tidal |iodels in making the student 
demand forecast. 

A cost - efficiency index was developed tliat combined the cost of running 
the model with the accuracy of the result. T/iis index measures the relative 
cost-value of the six models studied. 

This study will help institutions deterrhine whether the tfost of the. 
increased complexity and sophistication of /the forecasting model is compensated 
for, by an increase in the accura^cy of the student forecast used for budgeting 
and planning purposes. - 
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SECTION III:- PROCEOURE 

•\ ■ , . 

DESCRIPTION OF THE MODELS SELECTED . ' • - 

The models . sHected for the study wi^l now be discussed in their ordetx of 
increasing complexity. ' " • • 



' • ■ -. Judgement Only 

The simplest and most straight-forward method of forecasting is for a 
knowledgeable person to sit down and estimate the^ numbers based on his own 
good judgement developed through long years^ of experience. This "seat of . 
■your pants^' method sufficed in/the pasfand is still in use today in many 

' . V- 
institutions, 

, Even though the definition" of the procedure is simple, the execution' 
% task can be arduous.^ At the University of'Utah, for example, there * 
are 48 credit producing departments. When ,one considers 3 student' levels v 
for each department, that totals 144 dec^sions^. If one is looking 5 years i,r>.to 
the future, that explodes to ,726 individual decisions. Even thoug^h it is time 
consuming, the .task can be accompl isj^d. , However, one must consider the pro- 
blem of humane-fatigue and inconsistency when, estimating what the accuracy ,6f the 

final results wil.l - be. - • . 

' . Never-the-less, the Judgement Only Model is legitimate and must be con- 
sidere'd as a possibility, especially in times of cost-value. 

* . \ ' Basel ine - • ^ 

The B-aseline is the next degree of 'complexity" when considering fore- 
Casting models. The Baseline Model^ is an example of a ratio niofdel . This 
■ • " --13- - : 
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simple ratio model forecasts next year's student Jevgl by multiplying the 
base figure by a ratio. Only one ratio is applied td each leve> in eacjj^ 
department, but a different ratio may.be used for. each year in the projection- 
horizon. Stated mathematically the BaS<Hne Model becomes: 

•Yt+1 + Y*R, where Y^-is the SCH of a If^vel within a. d^artment, " ' 
,t is the base year and R is the universal ratio. _ - 

Th-e ratio- to use for each year is based on the estimated change in total 
university SCH from one period terthe next'. This universal ratio |s then 
applied to all levels and all department. ^'^^'^^ 1. 

This rnqdel could be made "more" complex by ..selecting a different ratio for 
each- le(^el,. 6Kfor each department, or for each- level in each department. How- 
everi the latter wbuld boil jdown to- a Judgement Only model if appl ied for just 
one year. "^If one select-^ different ratios "for each year, one again ds faced, 
with the hutnan fatigue facjtor. ' , 

. , Exponential Smoothing 

■ The'.Exppnential Smoothing Vdel was selected to demonstrate the fore- 
casting ability of a simjDle mat,ttfematical technique. This rnpdel is an Sample 
of a moving average technique and'is- slightly more complex than the'Baseline 

Mgdel. , . 

Although ho reports of .u^ing exponential ^moothing for projecting- student 
demand .by leve\ was found in the literature, Lny examples of applying this 
techtiique to projecting other ttme series dataware ft)'und._ One such Example 
• is the use of exponential smoothing to foreca stales, -3' ^ 

■-.j»,IJie:-E^(-ptj,nential SmootHnng Model' requires historical data because It 1s 

basically a moving average technique. ' The Exponential Smoothing Model steps 

. ■ -14- 
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d average tr'end of 
val ues . - 



thKjjugh each histoVical year and cal culates/^a- we'ightedjaverage and_ajrend 
usirjg the. exponential sfnoo'thing teChniqu^ devel oped byiRo,bert G. Brown. ^ 

A " * * • / W.. • ? ^ tiff 

. This technique calculates a mo^irj^ ayerage by usjfi| the weighted average^ 
of all past observations together* v^Wthe current val le as the basis for 
predicting the next' and subsequi^t values. The weight; 
;the data is also considered i/ predicting future data 

The deterininatian of hdw much^wejght to put on past data versus the 
cur-renj: data point in pre^licting the next value is handled by smoothing -con- 
stant, a variable with a value between 0 and 1. When the smoothing constant 
(alpha) is near zero, 'the historical data has^'the greatest weight. When 
alpha is near one, the curr^t data has the greatest weight in predicting 
'the^next value, - - ' , ^' ^ ' ^ |, 

The equation for calculating alpha is as follows: 

alpha ='2/(N+l), where N is the'number of historical 

' ' f ~ ■■' ' ' 

data points to use, in determin^'the*hext predicted value. ^Examples of the 

relationships between alpha and N are summarized below: 



N 



^Ipha 



2 
3 
4 
5 

10 
20 



.•666 " 
^500 
0.400 
0.333 
0.182 
/ 0.095 



The smotfthirjg constant can be set to have any number of years as 1:he ■ 
basis for -the^rediction. r The alpha thaf^rcvided the best results for this- 
study put" the greatest weight on the most; rec.ent .fotjr yeafs of data or on 
alpha of. 0-.40. This is cp^sistent with an earlier test a> the University of Utah 
where exponential sjno^inp was lised to predict the total university enrollment 



for short periods. 



■T5. 



*The exponentialy^qothing formula is stated mathematical'ly as follows: 

-^t+i Yt + \]-<) It, whe»'e;Yt_ is the current SCH value," 
<;;^ is th.e smc/thing constant and zi is the current niovi\ig average. The trend 
of the da'taMs calc^a'ted using a silnTlar formula as follows: 



+ (1- -C ) Tt, wh€re ± is the difference 
between the updated moving average and th|^rrent moving average, ^ is alpha 
the smoothing constant, Tt is the current moving average of the trend and 
Tt+1 is the predicted^next value-foT^e trend. . . . 

The next predicted value for SCH by level' is calculated as follows: 

" Yt+1. ="Zt+l + [{1- '^)/'^ ] T't+1, where vWkis'the next 
predicted value. „ ' 



Li near Regression • • 
Linear Regression is one of the class of regression analysis models. This 
technique is more complex than the Exponential Smoothing Model and reqijires 
'a good deal of accurate historical data. ' ' 

; ; " ^In this model, the trend of the past data is fitted to a straight line 
■ by the method of least squares. It i? assumed that future values will follow 

I' • the same historical trend. The trends may be short or l^ng range. 

In this study, although ten year's historical data wg'relavailable, only 
. . the most recent four years of SCH by level by department were used in the - 
regression model . 



■16- 




ERJC 



'22 



Combination - Linear Regression.' Ratio and Judgement - 

" . • ' ^ 

The University of Utah Model is a combination modet^ as detailed'in a 
previous section. It begins, v^ith two linear regressions, 'adjusts th-ese by^ 
*a ratio to meet 'an exterfially determined limit ba,sed on several causal /actors 
These projections may also be tempered* by human judgement, experience and, 
special knowledge. These judgement factors are, used on an exceptioli basis 



/ 



only where deemed necessary. • ' ' 

The data 'from the University of Utah model is-labeled in this report as 
the"University Model."* When the results of the University Model output' 
V/ere modified by inserting judgement into the"" model for seletted levels in 
selected departments, the data is labeled the "Special Knowledge" Model. 

Markov „ • <■ . 

As explained e^rli-er, the WICliE Student F]ow Model is basically a Markov 
model using transition probability matrices, and Markov chains. This is one 
jf |he most complex model s- currently in use,, using data edits, historical 
[analysts of'-^Ais^admissions criteria a-nd transition logic. The SFM system 
uses ;.17 "f)rpgrams ancj 14 sorts" to complete its task of predicting student 
■demand" on departments by level. 

Detail descriptions of the system', t|ie inputs required, the processing 



logic affd the reports are found in the WICHE documentation: "Introduction 
,to ,the Student Flow Mod^l SFM-'lA,"l*3 "Student' Flow Model SFM-IA System 
Documentation," ""^ arid "Student Flow Model SFM-IA Reports. "^^ 

In a nutshell, the SFM-IA system requires a file of studehis enrolled 
at the University ^for two time periods, t^ie FalT-QOarter efiro^^^^ w^re 



^^^^ 
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'seTectecKfor this study. Fron these file's it '•was\d-etermihed hovf man/ ■students, 

\■^Qre /ev from 'one period to the' n.e:xty Irowm^ny stud.^t5, we^^'exo 

at /hich level and, department , arid how many^ stod^n^s le.^^^^^^^ 

e/ther by graduation or by drop-out/ tijfs/data; was used"bi;/-t^^^^^^^ 

generate the transitton probability matrix -wMch. sVecifies what,^ of . j •. 

sjttfdents ' f rom one year move' to. a'. different department _^or 'level the nej(t; - ;• .,.>':i' 

year and what" fraction °sta.ys':whe:ra they are for the'-nefxt year . „ ^. . v,.,, , 

In addition to the transition Vrobabilit^^ matrix, the^.SFM-IA also requires' , 
the' starting-enrollment for each department, and level together v/1th the. . , _ 
expe'cted -number, of. new; sitlcients .of various^^^ c i .e. , freshmen, tracts-. ;. 

. - ■ . '■■ ' ■ ■"■ , ■ ■' ■ . - ' 

fers, returnees, etc.-. ■ . ^ ■ •■ . . 

# ' " ~ ■■ •" * -'■ • 

/5 summary of the eharacteristics -of €ach model "is shown in. Table I, : 

Summary of Sttiderit Forecasting Modelf. . In addi,tiQn^,to the model name and . 
type are some indicators of the 6bB^pl.exity Qf the, niod^^ ~ 

The Data Required tens how exten-sive a fi*e must be pulled together to 
operate the .model. Averag^Cost to.. Run and the Number of Programs refer to the ■ 
computer programs and the cost to execute the. system fpr one year's projection. ^ 
These costs are the actual cost on the -UMIVAC 1108;system at the, Unij/ersity of Ut^h 
" The Control Parameters show what the' analyst has <:ontr0l ov^r when 

using the model., -The parameters range from ^exactcqntrol over each number ■ 

in the Judgement Only Model, to control over a single parameter in the next ; 

three model s,^to control of several 'parameter^ that directly effect the ■ / . - 

results o"f the moderfofecast 'in the last three models. 

, ... ■ - . ..." ■ \ 
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1 -Inigl ementatyion 



^ As was mentioned earlt^^^ thei'ini^J^yf 'of Utah modeVwas already. .. ; 
leve),ope4; It was wmh^)l '^i^^ FORTRAN program and subroutifles ■ 

for the UNIVAC 1108 111 1968. Thg model was extensivgly modi,fied froml ' 

v| ■ . . ■ ■■■■ ■■,.<", . . ' A- 

igfegko 1971, but has retnained fairly -'stable since that tip'. The mod^ is .. 

/used by the Unjvers\ty several ' tim4 eacW-year to make projections for b\jdget 

"• pl-anning and resource al location./ The data from the system is 'used through- 

out the:year. / 

The data from the Dniversity Model and the .Spetial Knowledge Model are 
by-products of these annua? runs. made by the Office of Academic & Financial 

Planrtingj- ''■.--/ ^ ■■ " ' ' . ' 

The Judgement Only Plodel,. got it-s-data Vrom the Director of the Office 
of Academic and Financial Planning,. D^jring Fall Quarter each year he ,pote 
down his experienced judgement of whafthe SQH for each.level of >each depart- 
-ment would be for the academic ,year.; He Is per%s'the best qualified 
^person.dn campus fpr thfs task because he- is^lntimatel^ i,nvol ved .wiffh- the . 
■ "-budgets and planning of all academic' departments Sf the University. _ 

The Basel i-ne Model, the Exponential Smoothing. Model antd tJ^e Linear 
Regression Midel were written especially for ttiis^ prodect. They -are all in, 
' one program that has access to the same ten year hist|rica1 SCH data base that 
■ is used by the University l^odel . The FORTRAN •progr'^ms'j.and sutDr^tjtines were . 
/ all thoroughly 'tested and~ checked-out prior' to the ex-perimental run&. 

Several choices of the control parameters were' experim^ented with before 
' the ones final,ly .used\ere selected aS- the best -for the particijlar forecasting 



task. ' 
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' The WICflE SFM-IA Model caused the most |>rob1ems, used" the most time 

ana*cos4; the fnost m'oney to develop of any of the models. The; maj-or pro- 
•/ • ... 

blems encountere'd are summarized in Appendex I, Implementing the WICHE ' . 

Student Flow Model on the U^flVAC -1,108. - , 

The SFM-IA Model also 'iisfed a different data"^ base than thfe-other models, 

requirihq the file of ?tudentr~enrolleci^t .the University for.four consecu; 



tive Vears. ■ These files were availajj-l^but had to -be 'converted to be used 
on.the.UNIVAC 1108 machine. A speci.a.V COBOL program was written tt take 
\fiese files, in consecutive pairs and create^^hr student records specified 
by the SFM^Ia Mqdel . ' „ , . 

,^ V«The SFM-IA Modal was imp-lemented on' the 4JNIVAC 1108 antf was che^ked-out 

using the test (jlata .and reports provided by. WICHE. All ,17 programs and 

* ■■ ' * ■ ' ^ . . ■ 

- reports checked-0ut with the published results. 

.... " " . , * ■ ■ 

- • Experimental P]an 

The' research plan was to run each model using t4ia . data" for three 
academic years/ The projected SCH was to be compared to the actual SCH for ^ * $k 

that year by 'department and by level. * ' ;\ . « •. ' " 

. \ r ' - , • 

\ ' ' ■ V' 

Da t^/Ana lysis ^ • ' ' • : 

■■ ■■ ■ ■ ■ . . - * ' ■. •■ ^ ■ , - 

w / . " ^ ■ . ^ - i ^- * ■ - ■ 

^- A' spepiaV-data ajlalysis program' was wfitten.^o subtract the- ac«ua^ SCH ^ 
^'n^om^ the^roj-efcted SCH by department% level aad to examine the difference \ 

or the er^or. ^Several Statistics we^e calculated using these d iff erencfes,,^ 

„ * ■ " ^ ^ * . ^" ■ * - ' ■ . * - • ' 

• '.^ * * " , ' " * ' ••• * • 

including the followin^^: / - .-t • * ' T V -^ . .'^ • ,1^ C 

, ■^ * ,•• ^ ; ^ ■ ' • ^ - ■ - 

. , a. Mean Square Error;-" ; . ^ ■ v 

bl Mean Absolute Error ^- r . ^ ' 

' - c/^Mean Cumulative Error • . - ' - . 
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The Mean Cumulative Error gives an estimate of th6 averjige bias of the 
projection technique, i.e.., is the projectionValways over or always^der 

.the actual -value. . ^ . f 

" ■ • ^ • ' • ' 

The Mean Absolute Error gives an estimate of .the average de^fee of 

project-ion errar; - ' ' ^ . 

The Mean Square Error gives ^estimate of the 'average variance of the 
projection. * . _ ., * - 

These differences for each department were summed into coll e§e5\'3LVi^l ^ 
into the total University where the above statistics were calculate^cl ;.forv the 
collqge and' for the University by level., ^Figure 3 shows, an example of the//:, 
cofnputer printout fpr the TOTAL UNIVERSITY using the UnivfeFsity Model for ^ 
Acac(pmic Year 1975. Each of the above statistics is shown GkrcUl^t^d :f^^^ 
this medal . ■» 
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SECTION IV: RESULTS 



.. .. /, 
ANALYSTS Qf RESULTS 



Mean Square Error 




The statistics from tjie various cl|ita analys^^ computer runs are sum- ' 
marized -in tables in- Appendix II. T^bVe A shows the Mean Square Error for 
the^total University wheh summed. by departments^ Jhis/tabl^Nindicates that, 
the'Baseline and ^Special Knowledge Mo^Hs have-- the smallest erro't^^r^ dif- 
f^rent years, v . ^ y 

.'.Table B shows the Mean Square Error by level Yor the total University 
when surrftiied, by departments. This table^ indicates that general 1y /the errmr 
'is larger with lower division than upper and is larger with upper division 
than, graduate. The Judgement Only, BaseTilne, Exponential Sfnop/hi^^^nd 

three y^ars.^ ' - . 

^ Since the Mean Squcire Error values '^don't really mean/much by themselves, 
it is enlightening to look at^ them relative \4>* ^ standard. Choosing the Judge- 
ment Only Model as the standard, table C shows the factor that each model 
is of that standard for the ^otal SCH^ for the total University. The Bas^eline 
Model is consistently lower than, Judgement Only as i^ the Exponential Smoothing 
Model. " 

. when'^the factor is taken on the Mean Square Trrbr data by. level, Table 
is the result. This table shows that except for the fol lowind^ruimber of 




times, the Judgement Only Model hc\d the smerUest mea-n sfluare error out of' 
the nine independent projections for each model: 



Model 



Basefi^ne 
ExppjneWlal 
Linear Regression 
Univer^si ty 
Special Knowledge 
WICHE SFMrlA 



Number^ of Times out of Nine . I 
that Mean Square Error was J 
less than Judgement Only Model 



2 
2 
3 
0 



\ 



To try to get a feel for the pverall comparative effectiveness of the 
models in relation to the Judgement On^Model , a product of the factors was 
taken. ^Table E summarizes the'r^stjlts from both Tables C and D/ 

When, the three factor s^-^or each model are multiplied it produces a num- 
ber that indicates the relative s^'ze'of the mean square error of the model 
v/hen compared to the' mean s-qiJar^ error of the Judgement Only. Model as shown 
in Table^E,^ ^IL mAdTiTarfe'tnuch greater than dr ja>e fairly close to'the Judg^- 
men t Only Modef except for^^the following: 




Model 



Ba^l i n e ^ 

E;(^onerftial Smoothing 
^bpfecial Knowleclge/ 



=:>«Rfelati ve §ize of Mean Square , 
Error with Respect to the Judge- 
ment Only Mcfdel ---yTotal SCH 

{ ' . , 

J3 



<^When the- three factors' for each\leve1, of each model are multiplied 



together, the^roduct^is a.' number that) rbpresei^ts the relative magnitude of. 
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the mean square error with respect to the Judgement Only Model/' These values 
are shown for each Model in Table E. Mote that Baseline and Exponential 
Smoothing both tiaVe two values less than 1.0 and Special Knowledge has one. 

To produce a sirrgle number that represents the relative value of each 
model, thefour products in Tfible E were multiplied to produce an overall • 
factor. This factor measures the relative siz^^of the mean square error of 
each model when compared to the mean square error of the Judgement Only Model . 
A'value less than 1.0 would mean that most of the time the model's error was less 
than the error of the Judgement Only Model. The larger value would sh6w the ^. 
degree that the model's error was greater than the error of the Judgement 

Only' Model. ' " . ' ' 

Based on this overall criteria, the ranking of the models. would .be as 

follows: .> 



ModeT 



Rank -Overall Fabtor (Table E) 



^Baselin^ _ , 1 " 'Iao ' - ' 

Exponential Smoothing 2^ \ ^ 

Special Knowledge 3 v838 

Judgement Only 4 - 1.000 

University 5 3,947 

WIChlE SFM-IA ' 6 • 54, 777 

Linear Regression 7 * 66,948 

In addition to the ipean square error, several other measures of model 

it 

accu^ac^ wera used. The following paragraphs discuss the mean cumulative 
error, f he mean absolute error and^he percentage -error. 



^ Mean Cumulative Error " ' 

' The erpor of the projection for each level by each model was calculated 
• by taking the promoted vaMe and 'subtracting the actual value, i.e., 
o Error = Projected - Actual 
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• An average of these ermrs gives a measure of the degree of bias in 
th^ model.. This average'is the mean cumulative 6rror.° If the mean cumu- 
lative error is always negative, thiV means that the model is projecfing -^j. 
smaller values than needed. If the mean cumulatiye^error is always positive, 
this mearvs that the model is projecting val ues 'larger than actual too much of 
the time. The ideal is to have small ^valki^s for the mean '^cumulative error 
that are about equally po*sitive and negative. . . • 

• Table'F is a summary of the' mean cumulative error for'the model's in this 
"exp'erirfient.* The'-average absolute value. o,f these errors, the. ayqrage possible 

errors, the a'verage 'negative errors, 'and 'the number of positive and negative 
values is^g^ven below; ' , 

Mean Cumul.ati ve Error 
Model Positive Negative %. Posit jve Avg. Negative- AW.^Afa^sol ute 

^ Judgement 0.ly^ ' 5 ^ J 342,5 ' . ' ^ ^^-^^ 

Linear degression 5 . " 4 • 5B6.0 . 22Q.3 423.4 

University 5^4 237.0 -211.3 225 6 . 

.Special Knowledge, 5 4 304.0 < ' , oJ^? 

WICHE SFM-IA 5 4 506.6 187.5 254.1 ^ 

. ' ■ : . 

The above table shows that al 1 of the^ models are evenly matche^.with 

> positive and negative values, therefore no model bias^^^is suspected. -How&ver, 

the average positive values areij-igher, than, the aver^^e negative values for . ^ 

each model indicating a tendency to forecast high.. * 

T,he average absolute values would rank the models, as follows: 

Model Rank ' . Average Absolute Mean 

. ^ Cumulative Error - ' 



Special Knowledge 1 . - 216.3 

Judgement Only 2 • 219.3 



University „, 3 

WICHE-IA. ^ ^ 4 

Baseline 5 

Exponential. Smoothing " 6 

Linear Regression 7 



225.6 
251 ;1 
291.6 
- 307.7 
423.4 



Mean Absolute Error . . 

- The mean absolute error is calculated by taking the difference between 
^the- projected valye and the actual value by level and then by averaging the 
absolute value of these differences. This statistic gives a measure of the 
degree oferror of the projection from the actual. 'The advantage of this 
statistic over th«mean cumulative error in measuring the magnitude u©f the 
projection fe^rror is that in the mean cumulative error a large negative error 
and a large pQ^itive error would cancel- each other out and make tt^e, mean; 
cumulative error lo&k small*: In. the mean absolute error, all deviations from 
the actual value are taken into account. 

Table G is a summary of the mean absolute error for the modpls in this 
experimeVit. The average mean absolute error for each modp^l is given below: 



(IV 



Model 



Judgment Only 
Baseline 

Exponential Smoothing 
Linear Regression £ 
University ' • 

Special Knowledge »" , 
WICHE'SFM-IA 



Average Mean/Absolute Error 



/812.3 
?738.8 
787.3 
993; 1- 
822, 2 
769.2 
967.8 



The -average mean absolute error values would rank the models a.s follows 
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Model 



Rank 



Average Mean Absolute Error 



. Basel ine 

Special Knowledge ' 
"Exponential Smoothing 

■ Judgement Only ' , • 
University 

WICHE SFM-IA 

■ Linear Regression 



4 
5 
6 
7 



738.8 
769,2 
787.3 
812.3 
822.2 
967.3 
993.1 



Percentage Error 

The percentage error reveals what percentage of tHe Mtal value 

^ r 

the error is, e.g., for lower division SCH 1n the accounting depydrtmenls , the 
error -Was" 2% o.f the lower division SCH for that year.. Ttiis measure of error 

- ' • \ " ^ ^ ^ ^ ^ . '- v ■ 

is valuable because it reports the er^ror relatiVe to the^ctua.l values.- An^ 

' • . ^ ' ^ ^ X ^ 

error of 4 compared to a value of 10 is mucji' more significant than an en\or 

df 4 compared to a value of lOOw, (The <iirference is '40% errpr compared wit) 

4% error) . * . ' ' . - 

Table H is a summary of. the percentage error for the models in this^ 
experiment. The range, of error for ±he fnodels' goes from O.Vl% to 37%. The 
table quickly reveals that all the models ha*d trouble predicting the 1975 
grajidate level SCH. Ei^cluding these ^aJ'uejB, as outlj^rs, the average^ 
percentage error for, each mode\ is given below: ^ ' . 



Model 



Judgement 0?ily , • 
B£fse,line ^ 
Exponential Smoothing 
Linear Regresrsian 
University , 
Special Rnowledge 
WIGHE SFM-IA 



Average Percentage Error 

/ 



\ 



4n68 - 
5.43 
5.52 ■ 
7.40 
3.00" 
2.74 i 
8.54 
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The average percentage error values would rank the models as 



f ol 1 dws : 



Average-perj^eojbage Error 

• 2.: 



t 

J 




Model ' "Rank 

■7 • ^ 

Special Knowledge 1 

"University ^ . ' 2 

Judgement Only 3 

Baseline 4 ' 

-Exponential Smoothing * 5 

Linear Regnsssion 6 

WICHE SFM-I/V • 1- 



CONCLUSIONS • , ' " - , , : 

-'Vhe purpose -of this, study was to. compare the ac'curAey of the forepaU 
of' student demand' by level^on academic departments iltsing six basic methods.. 
C'and to select the' best' model. The -study was also to^determine if. the cost of 
the more complex models can hjs justified in terms o.f increa^^ed accuracy .of the 

forecasts. - . ' ■ ^ ■ '•, • . 

The results of the study presented in the previous section are summar.i'zld' 
below showing how each liiodel was rar>ked by the various error analysis, criteria 

Model." " Mean Square Hea 



Rank 



Judgement On ly-> 
J&fn?feline ' 
Exponential; Smooth ii^g ■ 
• Linear' Regression 
liversity 
sciaT^ Knowledge 
fCHE SFM-IA 



4 
1 

2 
7 
5 
^' 
6 



2 
. 5 

6 

7. 

3 
. 1 

4 







4 


.3 


. 1 


4 


•,3 


5 


7 - 


. 6 


" ■ 5 • 


. 2 


2 


. ■ 1 











To -get a feel for the pverall rank, add'the- rankings above and compare. 
The (results ^re shown below: . • ' 



1 



Model ^ Accumulated .Kank Summary of Rankings 



Special' Knowledge ^ 1 7 

Baseline . ^ 2 11. 

Jii?dgement Onjy * * * 3 15. 

Uriiversity ' . , - ' . ' 4 . .15. 

Ey^ponentiaT Smoothing 5 '16 

WJCHE S'F.M-IA ^ ^ 6 . ' ^ 23 

Dfnear Regression * v 7 . ^ 27 



• ''The above tab'le suggestCs three groupings, for the models. The first, is the 
Special Knowledge Mod?! which* appears superior to. the others. The second 

. grouping is the Bas'elnne Judgfement Only, University Model ^and Exponential' ' ' ' ^ 
Smootlring where there seems to be no significant difference between the mocPels. 
The 'third grouping includes the WICHE SFM-I-A and Linear Regression which 
appears to'be infeHor to the other' models studied based bn the chosen criteria. 

To determine the r&lative. cost-effectiveness^of the models, an index was; 
developed which is the .p^roduct of a measure of effecti vene.ss and a .measure^'of ^ 
cost. The effectiveness measure phosen was the average of the by-level mean 
square error data. This is' the' prime accepted measure of error for the study. 

The measure of cost selected was the computer operating cost. It is 
recognized that all the'models require some hqman time to 'set^ upland prepare, 
the data. for a-run.' Even the Judgement Only Model requires some human time.. 

, Jt a^5ume^-that for tlnESsjiiodels the_ human time wa,5hs,not significantly 

,■ different among the mod^els and could, be eliminated from the selection criteria. 
Jhe remaining cost Js the computer cost which can be obtained directly off 

' the ruil sheet Snd is, thus, readily available. - 

The average ftiean square error for each model and the average cost are 

'summarized in Table, I. The calculated cos,t-effectiveness index is also shown 
there. 
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H6de1 Rank ' ' " Index 

Baseline r • V , ' ^ „ ^ 

Judgement Only 2 " . ' .1.79 

Exponential Smoothing , V ° 

Linear Regression 4" ' / 6.03 

Special Knowledge " -5 — ^ ' . ^ * ^ 32*75 

University % ' . 37.00 



WiCHE SFM-IA ^ -'7 ' ^ J 1,901.42 



' 'From' the above table, the WICHE SFM-IA model is definitely not a con- 
-tender in'the cost^^effective'ness contest having both-a relatively large 
error and a' large cost. ^ .i. a 

The conclusion of the study, then, ns that a model ^that combines the ^ ^ 
objective extrapolHtion mth the causal expect judgement produces the best ^ ^, 
■ results .in forecasting the student demand on academic departments by level ^ . . 
as demonstrated in the-. Special/Knowledge -Model . 

The study has aTso shown jt^hs't simple models provide just as good if 
no^t better forecast's tfia^n do complex models. ' If the cosj^t of executing the' » 
model fs considered, the simple model should be chosen over the complex one-r 
every time. ^ , • a 

• ' RECOMMENDATIONS - * ' " ~ . ^' ^ ' . , ' 

The research reported here showed the^'value of combini,ng expert judge- ■ 
• ' . " ' '/ ^ •• 

~ ment with ma.theniatical • model s in foreclasting student demand. Jhe, research 

also pointed out the valtie"of simple mcKieTs. , ' 

The next logical step to experimental th is to use expert judgement in . ' 

■ some simpler models such as Baseline\nd Expon,e?itia,l Smoothing. -Earlier^ in 

the report it, was suggested that li^he univers&l factor in the Baseline Model 

could be replaced by different factors for each level or by different factors 

-32- 

ERiC ^ 38 ' " 



" for each departirient.V Per haps different factors c&uld be used for selected 

> depaftfnents.or 'se^^^^ done in the Special Knowledge Model • 

/ X Quite di bit of. work has- been .Mne with the Exponential Smoothing Model. 

, where the tdntr^^^^^^^^^^ alpha, the srrjbothing constant. Several models 

have been devel#ed. to chanle aV^^^ 'based on certain calculated criteria, 
Perhaps a niQdel cogltf; be^c^e^^^ to change alpha based on expert judge 

, jDjentt vf or. certain ^dep!^^ " ' - 



o 
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APPENDIX 



•I. Implemefiting the WICIIE Student Flow Model on the UNIVAC. \n08. 



II. Data Tables 



A. ^Mean Square ^rror for Total ^CH of ^the Jotal University SuDimed 



by Department 



■B. yie^n Square Errpr by Lev^l fbr'SCH of the Total University Summed 
by Department. ' . 

C. Factor of Each Model's Mean Square Error for tptal SGH of the 
Judgement Only Model. / . \ ^ 

D. Factor of each Model^'s Mecir> Square Error for SCH by Level of the 
, Judgement Only Model . 

E. Product of the Factors in Tables C and D for Three Yiears. 

'Fi Mean Cumulati.ve Error Level for SCH of the Total University 
Summed by 'Department. 

G. ^Mean Absolute Error by Level for SCH of the Total University 
Summed by Department. ^ 

H. Percentage Error of SCH Prdjectior\ for SCH of the Total University 
, Summed^ by Department. " - 

I. Cost - Effectiveness Index for Student Projection Models. 
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' > . APPENDIX I 
&W£f^N&^-H£JlI^ErST FLOW model' ON THE UNIVAC 1108 

The WICHE Stydent FJow Model was implemented on the University of Utah- 
UNIVAC 1100 computer in .the Spring^of 1975. Several actions are require of 
the systems analyst to get the series of programs up and running. Some of V 

these actipns were anticipated-and documented by WICHE and some were unexpected. 

' - ' »< ^ 

Below is outlined^ the types of problems encountered in getting .the 17 
ANS COBOL programs oi" the Student Flow Model to run on-,t/ie UNIVAC 1108 machfne: 

• . ■ • 

I. Softv/are and documentation delivered late. ' 

ir. Actions documented by VMCHE ■ • ' 

■ . - • ■ ' ■ . » „ . , ' ^ ■ . .. 

A. Modification, to the ENVIRONMENT DIVISION , • ° 

B. Reqi^irement of 14 sorts ' 
III. Unanticipated' problems ' ' * 

"A. Corrections required to get a clean compile . ^ 

* 1. TDP-OF-PAGE . ^ 
, ' ^ ^ 2. REDEFINES ' \ \ . . ' 

B.' Corrections required to get an error-free execution 

1. Improper' label ilig of sort control fields- 

2. Referencing data items fn closed files . 

3. Remove Segment Feature from some programs. 

A detail description of each of the above actions required to get the system 
running is given below: ^ , . • 

L Software and Documentation Del ivered Late. The NCHEMS Student < 
'Flow Moddl SFM-IA was^ originally scheduled by WICHE to be released in October 
1973. M received a copy of the "Type II : NCHEMS early Release Programs" artd ' 
"Preliminary Draft for Review Purposes Only" ^system documentation in mid- , 
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Appendix I (continued) 

September 1974, almost one full year after the promised ^ate. The difference 

between the original and the actual delivery date has raised havoc with .the 

scheduleid completion of this project. * u 

In addition to the late delivery, thes*e "Early' Release- Programs" contain . 

a warning that/ "software will be programs that have not yet been adequately 

tested or documented for release* as Type I software." The following? paragraphs 

discuss in detail the problems encountered using these programs. ^ Tracking' and cor- 
recting the errors , caused at least a two month delay in this project completion. 

' ■ - \ t • 

II. Actions Documented by MCHEMS- - ' 

A. Modification to the Environment Division . Part of the WICHE documen- 
ta^tion warned that certain changes must be made in the ENVIRONMENT / 
DIVISION, l^e following are^examples of cards that were input to ^ 
correct each of the 17 programs. 

-SOURCE-COMPUTER ^ ' UN I VAC - 1108 ' ' 

OBJECT-COMPUTER UNI VAC - 1108 " ' 

\ ° SELECT REPORT-FILE ASSIGN TO PRINTER 

SELECT OLlD-SFMrFILE ASSIGN TO UNISERVO NEWFOL. 

..1} v.^- , T- , • ' 

''select user-main-file assign to card-reader " ' 

B. Requirement of Four-teen Sorts . The SFM-IA system requires the user ■' 
to supply 14 sorts to operate. A standard UNIVAC 1100 STANDALONE 
SORT/MERGE package was used for this task. Following is an' example 

of the sort control cards: . , . . 

0 RUN 

0 HDG SORTIO STANDALONE SORT HISTORY MODULE . 
(3 ASG,A NEWF10.!^F2,. ' ' 



0 ASG,T NEWSIO., F2 . . 
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0 UUCC*. SORT Mf RGE. SORT 
FfLEIN ? NEW no 
KEY = 1, 37, A, A, 1 
tiLEOUT = NEVfS 10 • 

0 FIN " 



III. Unanticipated ProWems. - • 

In addition to the above actions docufdented by NCHEMS that are required 
to ge^ the system operating, several actions were required to ge.t a clean compile 



and others were required to achieve a successful execution of the programs. 

A. Corrections Required to get a Clean Compile . , The following" cards were 
, ' changed'to get an error. free- compi le using the^@ ANSCOB compiler on 
the UNIVAC 1108: ° 
TOP-OF-PAGE • 

'oil owing card from the • - \, 



\ 



1 



Delete the 

'ENVIRONMENT DIVISION:- 

SPECIAL-NAMtS . IS TOP-OF-P/\pE. " \ 

Add the following card to the DATA DlVIS^lON: ' 
77 TOP-OF-PAGE 77 PICTURE 99 VALUE 60. " ' 
Change the following card in the PROCEDUREI DIVISION; 
' . WRITE REPT-RECORD-OUT, AFTER ADVANCING TOP^-OF-PAGE 77 
2. REDEFINES . . 

' In' programs SFM- 40, 45, 50, 70 and 75. 
An 05 level I^EDEFINES statement did not reference back to the 
original 05'level data name but instead referenced a data name 
which was a redefinition of the' original 05 level, data name. 



Appendix^ I (continufed) 

The foil ov/ing is an example of the changes th^at were made to fix 
this' probl em flagged by the compiler: 

•' " 05 OLD-SPPI-DATA ' ' REDEFINES OLD-SFM-DATA ' ' 

05 NEW-SPPl-DATA . REDEFINES "• NE;W-SFM-DAIA 

B.. Cogrections Required to Get gn Error* Free Execgtfon . Once a clean' 

compile wa? achieved, more energy was expended tryinfg to get^a suc- 

*■ • . ■ . ■' ' ' 

cessful run using tfie vilCHE t-est data set. Following aire examples 

of this kind bf problem; " " ' ' . . 

'i . Improper Labeling of Sort Control Fields : NCHEMS mailed: out. a 

program change in December 1974 updating program SFMOl ^by fhoving 

spaces to MEW-SFM-RECORD. However, when ^fexecuting Sf;M02 after " ; 

^ ' • * ' . 4. 

S0RT02, a vital' control record was flagged as missing alid SFM02 4 
aborted. This record is the one' that assures that SFMOl ^xe-. 
Guted properly. ^ ' - ' , , 

It was found that the record was. ^created by SFMOl but it was 
located in the file out of .the sort sequente expected by 5FM02. / 
The NCHEMS program change put spaces in the sort control fields 
when the system required that vital sort data such as "iteration" 
code and "term number" be in those fields. ^ 
. ' Once these data were inserted back in the records,, the sort (SDRTOl) 

pullqd all the control records tp the front of the file where pro- 
gram SFM02, expected them to be located" SFH02 now execute properly. 
2. Referencing Data Items -in Oosed Files . In programs SFM20,o 25/40 . 
, ^ 45, 55, 60, 70, 75, and 80. the program referenced a ddita name ,tn a 



prior to the file being opened and It also referenced a data., 

name in a file after the file was Closed. Both of those condi- 

' ■ - ■' # 

tions caused program e)<eGution errors and aborted the run. The 
^ograms were changed so that the UPDATE-^FILE in SFM20 and the 
QLD-SFM-FILE SFM25, SFM40, SFM45, SFM55, SEM60, SFM70, SFM75 
and SFM80 are opened first thing in the program an<l closed the. ^ 
last thihg.' These changes allowed a successful execution* of these 

progr&ms. , ^ ^ <^ ' • ' 

Remove Segment F66h:ure. In several progra.ms myst^ious execution 

" • • ^ ' • ■ . : I. 

errors were occurring that the programming and systems people of . 
the University ^could not explain,. After the segmentation feature 
was taken off these programs, they executed using the WIC^^ test 
data and printed the same reports as> the doctimentation illustrated, 



WENDIX II 



DATA TABLES* 
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TABLE A . TOTAL .UNIVERSITY -'SUM OF DEPARTMENT'S MEAN. SQUARE ,E1^R0R* - TOTAL 



Academic Years /Proj e cted 




Technique 


1973 


>■ — 

|y/4 


1 07/1 


Judeement Only ^ 


. . 37.5 . 


41.. 4 


74.0 


♦ 

Baseline 


20.4 


• 23.6 


60.4 

> 


Exponential Smoothing - 


31.0 


34.5 


. 62.4 


Linear Regression 


80.6 


'40.9 


88.9 


University Model 


? ' 51.2 


, 30.7 


70.3 


Speciiil Knowledge / 


46.1 . 


29.9" 


36.6 


wiche'sfm-ia . \ 

V) 


56.6 


.? 44.2.' 


82-. 6 


Mjoctel with Small esjt Error^ 


Baseixine 

■ * \ - 


Basel llie, 


"Special 
.;, Knowledge 


. , ■ — 


** \ 


J* 
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TABLE B. TOTAL UNIVERSITY - SUM OF DEPARTMENTSj MEAN SQUARE ERROR* BY LEVEL 



Acaderoic Years Projected 



'Technique / 


1 Cii/cil 


1 Q7? 

1 I7 / 0 


1974 




1975 




Lov^er 
upper 
Grad 


24.9 
1.5 


16.1 
23.7 
"5.8 


r 


47.8 
18.1 
16.9 


baseline 


Lower 
upper 
Grad 


10.9 

0.0 

1.6 


. 19.4 
25.0 




39.8 
13.4 
22.5 


Exponential Smoothing 


Lower ( 
Upper \ 
Grad 


19.1 
/ . u 
^ 1.6 


26.1 
?1 1 
5.5 


A 

1 


37.2 
9 9 
18.5 


Linear Regression 


Lower 
Upqer 
Grad - 


53.3 
11.8 V 
.3.1 \ ' 


30.6' 
28.9 ' 
6.5 




67.4 
17.7 
26.5 


University Model 


Lower 
. Upper 
Grad 


27.4 N 
9.0 
1.8 - 


. 19.5 
\ 21.2 




49.1 
17.2 
27.4 


Special Knowledge 


Lower 
Upper 
Grad 


^28.6 
7. '9 
1.7 


18.2 
19.9 
7.1 




33.1 
17.4 
25.7 


WICHE SFM-IA 


Lower 
Upper 
Grad 


39.8 

12.7 " 
2.5 


28.6 , 
33.4 
8.1 




57.9 
20.4 
19.5 



Model with the 
Least ErP0r 



Lov/er 
Upper 
Grad 



Baseline 

Judgement 

Judgeiftent 



Judgement 
Knowledge 
Smoothing 



Knowledge 
smoothing 
Judgement 
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TABLE C. ^ TOTAL UNIVERSITY - SUM OF DEPARTMENTS MEAN SQUARE ERROft - TOTAL 



FACTOR OF JUDGEMENT ONLY BY YEAR , ' r 











Technifiue 


1973 


1974 


197^ ; 


Judgment Only 


. '1.00 


1.00 


• .1.00 ; 


Baseline 


.54 


.57 


.81 - : 


Exponential Smoothing 


.83 


.83 


.84 


■Linear Regression " 


2-.15' 


.99 


. 1.20, ^: • 


University Model 


^ '1.37 


.74 


.95- . . 

y 


Special Knowledge 


1.23 • 


.72 


.49 


WICHE SFM-IA 


» " 1.51 


1.07 


' 1.12 



,4 



. ■ ' ■ r 
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TABLE'D. - TOTAL UNIVERSITY' - SUM OF DEPARTMENTS' MEAN SQUARE ERROR BY. LEVEL 
' ■' . ' FACTOR OF JUDGEMENT ONLY. BY YEAR' BY .LEVEL ' < 

; , ^ • ' , '• . AcaderriTc Years' Projected 

' Technique . Lbyel ' 1973 ^ /. 1974 ' • ^ 1975 



Judgement Only 


Ldwer 
Upper V 
. Grad 


1 . 00. ' 

1.00 

1.00 




1 nn 
1 , UU 

1.00 

1.00^ • 


1 fin 
1.00. 


Baseline 


Lower 
Upper 
' Grad 


1.02 
1.07 




1.05 , ' , 

1.05 


^83 
.774 


Exponential Smoothing 


> Lower 
Upp^r 

%Aa . 


.77 
1.13. 
1.07 




1.62 
1.31 
• -95 


"'.78 
.55 
"1.09 


Linear Regression 


Lower 
Upper 
Grad 


2.14 
1.90 
2.07' 




1.90 
• 1.22'- 
1.12 ■ 


.78 
' .^8 
1.57 


University Model 


La^;er 
Upper 
Grad 


1.10' 
1 .45 
1.20 




1.21 
.90- 
1.24 


1.03 
^ .95 
1 .62 


Special ^^Knowledge 


Lower 
.Upper ^ 
'Grad 


1.15 
1.27 
1.13 




1.13 
.84 
1 .22 ' 


.69 
-.96 

. "1 .52 


-WICHE SFM-IA 


Lower 
Upp6r 
Grad^ 


1.60 
2.05 
1.67 




1 .78. 
1.41 
1.40 


1 .21 
1.13 
1.15 
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TABLE E. PRODUCT OF THE 


FACTORS 


FOR THREE 


YEARS IN 


TABLES C AMD D 




( 


Table C 




/■ ■ 
Table D 


/ 

A 




Technique 


Total • 


Level 




Product 


Overall 


Judgement 


1.000 


Lower 
Upper 
Grad 




1 .QQ.0 
,1.000 
1.000 


I . UUU 


Basehn6 


U , 

• V 


LUVvc 1 

Upper' 
Grad , 




438 
.793 
1.494 


.129 

< 


Lxponen ti a i onio,o in i ng 


n ^7Q 


Upper 
. ■ Grad 




.973 

/!8r4 
1.1Q8 


.508 


'Linear Regression 




1 nwPK* » 

Upper * 
Grad ^ 




3.171 
2! 271 
3.640 


66.948 


University Model 


0.963 


Lower 
Upper 
Grad 




1.371 
1.240 
2.411 


3.947 

♦ • 


•Special Knowledge 


. 0.434 


Lower 
Upper , 
Grad 




' -900" 
\ . 0^4 
>2.095 .. 


.838 


WICHE SFM-IA 


1.810 


Lower 
Upper 
■ Grad^ 


< 


3.446 • 

3.266 

2.689 


54.. 777 

- , ■ 


















• 






1 
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' "TABLE F. 'TOTAL UN^rVERSITY SUr4 OF DEPART^t|^lTS MEAN CUMULATIVE E^ROR BY LEVEL 
' ' ' . ^ (Errors =^ PFojected;- Actual) ^ ' , ■• 

V* . ^ . f i * ■ 









Academic Years ^Projected 




1 Tecrhniciue ' . " 
^ Judgemerit^ Only 


^ Level 


1973 


* .. • ■ .1 1 

' ,1974 


1975 


-Lower' 
Upper 

/H i 

, GracJ , 


, 529 
-201 ' 
-61 


112 , ' . 
■ ^ - . ' 261 ■ 

. . ^ - 1 JD 

o 


-47 ■ 
' -184 
too 


Baseligrie''^^ '\ , - 


' Xower* 
Upper 
Grad 

H 


180 ■ 

- -381 

- -69 


99 ' 
% > 333' 

-4oD 


-67 
^-299 
/ 1 U 


Exponent i'&l Smqothing. 


\ 

Lower. 
Upper 
Grad^ 


411v' 
-256 
•-4 6 - 


^ ' 520' 

» • ' -^^17—-'"'^ • 
: . -41 4 


9 

-165 

^"531 ■ , 


bi near^Regr^essi dp ^ 


Lower 
Upper 
Grad- 


' 873 . 
- 's-75 
V 268 


434 
' ; €40" • 
•,338 


" -377 
. -91 
.'715 


University Model ^ 


Lpw^r 

Upper 

Srad 
> 


^-168'* 
«-309 
' J"8 


' . 60 , * 
' 195 . 


-168 
-200 
777 


.Special Knowledge , 


Lower 
' Upper 
'Grad 


-112 

•-238 
,58 


52 

• 216 
43 


. -2B7 
- -114 
* 847 


: iflCHE SFM-IA 


Lower 
Upper 
Grad 


716 
-194 , 

113 , 


. ' 638 " 
37'4 
-134 


_235 

-187 

-,,^'692' 



TABLE G. TOTAL UNIVERSITY'- SUM OF DEPARTMENTS MEAN ABSOLUTE ERROR BY LEVEL 

* <f ■ ' 



■ t' ■ 

Academic Years Projected. - 



t 

Technique ' - . 


Level'' 


' ,1973 


1974 - 


■' 1974 


Judgernent Only ' ' 


' Low.er 
Upper 
Grad 


nil 

• 629 
. 27-3 ' ' 


1096 

1168 „ .„ 


12.01 . 
716 
620. 


feaselirie 

^ ■ .J. 


Lower 
Upper 
Gnad 


701' * 

■ "590 ^ « 
242 ' 


- 718 
1146 

.. _ 558 > 


. 1026 
a79 
789- 


Exporrential Smoothing 

• ■ « 


j> 

Lower 
Upper 
Grad 


992 „ 

622 

247. 


N 1045 A 
\ 1265 
1 515 


1042 
. 717 
641 


Linear Regression 


, Lower 
Upper 
■»Gra(t 


1522 • 
782 
395 


^1143' 

' . 1249 

,587 ' ., 


. 1509- 
957 
7,94 


tJniversitV Model 

■» ■ . 


Lower 
Upper 
Grad 


.988 
^ , "693 ~ 
293 ' '. 


812 

"r- 1090 
■ - 579 - <5 


.1112 
97B 

. 857 


Special Knowledge 

■: ■ ' ^ ■ 


Lower. 
Upper 
Grad 


895 
'711 
260 


789 
. 968 . 
461 


1021 
' 915 

'903 


WICHE'^FM-I^ ■ ' 


> 

. Lower 
Upper 
Grad ' 


1364 
68i 

382 • 


" • 1114 
. • 11.89 • 
' „ 614 


1486 
1012 
868 



TABLE H. TOTAL UNIVERSITY - SjJM OR DEPARTJ^IENTS PERCEtlJA^ ERROR 







Academic Years Projected 


Technique 


Level 


1973 


"si 974 


1975 


' - — ■ - . — 

Judgement Only 

«> 


Lx)wer 
Uppar 
GratJ 


6.35 
3.81 
1.48- 


4.75 
5.28 
6.19 


5.62 
■3.96 * 
10.11 ' 


Baseline 


Lower 
Upper 
Grad 


2.15 
7.22 

' 2,78 


1.23 
6.89 
^ 16.66 


, .81 
5.72 
31.35 


Exp(3|Jf^tial Smoothing 


Lower 
Upper 
Grad 


4.93 
4.84 
• 1.86 


6.44 - 
8.63 
14.19 


.n 

3.16 
23.45 


Linear Regression 


Lower 
Upper 
Grad 


10.47 
1.41 
10.82 


5.38 
13.23 
11.57 


4.55 
1.75 

» 31.56 


University Model 


Lower/ 
- Upper 
Grad 


■ 2.02 
5.86 
1.14 


-75. 
4.04 
" .4.29 


2.03 ' ■ 
, 3.83 
34.3b 


Special Knowledge 


Lower 
Upper 
Grad 


1.96 
4.91 
2.08 


1 . 62 ^ 
- 3.11 ^ 
2.87' 


3.21 
2.18 
37.38 


WICHB SFM-IA 


Lower 
Upper 
Grad 


(8.91 
«4.06 
9.37 


6.85 
15.12 

15.38. 

■ • 0 
o 


5.64 
3.01 
24.67 
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TABLE I COST - 

i f \ L_ X • \i/ V/ w 1 


EFFECTIVENESS INDEX 


FOR STUDENT PROJECT-ION 


MODELS 




Index = (Error) 


(Cost) 






Average of By- Level 


Data Mean'Square 




Technique 


Error Average Cost Per Run 


Index 


Judgement Only ' 




$ 0 (JO) 


, 1 .79 


Baseline 


16.1 


JO 


1 .61 , 


Exponential 'Smoothing 17.3^ 


J5 


2-.60" 


Unear Regression 


27 .3 


.25 ■ . 


5.83 


University Model 


' ' '20.0 : : 


,1.85 


. 37.00 


Special Knowledge 


vr • 17.7 


1.8"5 


32.75 


WICHE SFM-IA 




76.67 


1 ,901.42 


«' 


7 

a 
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